Repository navigation
fix(storage): do not apply retention or holds to parallel composite upload parts - #14894
Open
sandeepvinayak wants to merge 3 commits into
Open
sandeepvinayak wants to merge 3 commits into
sandeepvinayak wants to merge 3 commits into
Conversation
…pload parts Parts and intermediate compose objects were built from the destination BlobInfo, so they inherited its retention, temporaryHold and eventBasedHold. GCS then rejected the part cleanup deletes, causing the upload to report failure after a successful final compose and leaving undeletable part objects behind. Fixes googleapis#14892
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request ensures that transient parts of a parallel composite upload do not inherit retention or hold settings by explicitly setting them to null, as these properties only belong on the final composed object. A unit test was added to verify this behavior. The review feedback suggests refactoring the AssertJ assertions in the new test to use allSatisfy instead of stream().allMatch(...).isTrue() to improve error reporting upon failure.
Author
|
@ShreyasSinha I see you have been active on storage, can you please help to review it? Thanks! |
Author
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #14892
Problem
ParallelCompositeUploadWritableByteChannel#definePartbuilds every part and intermediate compose object fromultimateObject.toBuilder(). It cleared only crc32c, md5, and kmsKeyName, so when the destinationBlobInfosetRetention,temporaryHold, oreventBasedHold, every part inherited them. The final compose succeeded, but GCS rejects deletes of retained or held objects, so:cleanupPartsfailed, andclose()/UploadResultreported FAILED even though the destination object existed.Retention.Mode.LOCKED, they can't be removed until the retain-until time.Fix
definePartnow clearsretention,temporaryHold, andeventBasedHoldon parts and intermediate compose targets. The final compose target is stillultimateObject, so the destination keeps all three.Testing
New unit test
ParallelCompositeUploadWritableByteChannelTest#partsDoNotInheritRetentionOrHolds:maxElementsPerCompact = 2forces intermediate composes, so both code paths that calldefinePartare covered.The test failed before the fix (
AsynchronousCloseExceptionfromclose()) and passes after it. All 26ParallelCompositeUpload*Testtests pass.The optional follow-up in the issue (exposing
PartMetadataFieldDecorator/PartCleanupStrategythroughTransferManagerConfig) is a public API addition and is left out of this PR.